Glue Operators & Hooks
Serverless Spark, Managed by AWS
Glue is AWS's managed Spark — no cluster to size or terminate, just a job definition and a script. It also owns the Data Catalog that Athena (and Redshift Spectrum) read their table definitions from, and Crawlers that keep that catalog in sync with what's actually in S3.
Same honest note as the EMR and Redshift pages: a real Glue Job run needs a script staged in S3 plus a dedicated IAM role, and bills per DPU-hour with a multi-minute cold start even for a trivial script - enough setup and cost that it wasn't spun up live for a screenshot here, unlike the S3/SNS/SQS/Lambda/DynamoDB/Athena pages. The code below is correct, real provider code.
Running a Glue Job
from airflow.providers.amazon.aws.operators.glue import GlueJobOperator
run_transform_job = GlueJobOperator(
task_id="run_glue_transform",
job_name="sales-data-transform",
script_location="s3://my-scripts/glue/transform_sales.py",
s3_bucket="my-glue-assets", # Glue's own temp/logs bucket
iam_role_name="GlueServiceRole",
aws_conn_id="aws_default",
create_job_kwargs={
"GlueVersion": "4.0",
"WorkerType": "G.1X",
"NumberOfWorkers": 5,
},
)
GlueJobOperator creates the job definition if it doesn't already exist, then runs it and waits for completion — one task covers the whole lifecycle, unlike EMR's separate create/add-steps/wait/terminate tasks.
Keeping the Data Catalog in Sync
from airflow.providers.amazon.aws.operators.glue_crawler import GlueCrawlerOperator
crawl_new_partitions = GlueCrawlerOperator(
task_id="crawl_new_sales_partitions",
config={
"Name": "sales-data-crawler",
"Role": "GlueServiceRole",
"DatabaseName": "sales_data",
"Targets": {"S3Targets": [{"Path": "s3://my-data-lake/sales/"}]},
},
aws_conn_id="aws_default",
)
Run this after landing new partitioned data in S3 and before an Athena query needs to see it — the Crawler updates the Glue Data Catalog's partition metadata so Athena (or Redshift Spectrum) knows the new files exist.
Data Quality Checks
from airflow.providers.amazon.aws.operators.glue import GlueDataQualityOperator
check_data_quality = GlueDataQualityOperator(
task_id="check_orders_quality",
datasource={"GlueTable": {"DatabaseName": "sales_data", "TableName": "orders"}},
ruleset="""
Rules = [
RowCount > 0,
IsComplete "order_id",
ColumnValues "amount" > 0
]
""",
aws_conn_id="aws_default",
)
This is AWS's own managed alternative to running Great Expectations yourself — see the External Integrations module for the Great Expectations approach, which works with any data source, not just Glue Catalog tables.
GlueJobHook Directly
from airflow.providers.amazon.aws.hooks.glue import GlueJobHook
def check_job_run_state(job_name: str, run_id: str):
hook = GlueJobHook(aws_conn_id="aws_default")
state = hook.get_job_state(job_name=job_name, run_id=run_id)
return state
Glue: serverless, less configuration, per-DPU-second billing, best for straightforward batch ETL. EMR: full cluster control (specific Spark/Hadoop versions, custom bootstrap actions, long-running clusters), better for complex or highly-tuned workloads. Most new, simple pipelines should default to Glue; reach for EMR when a job's requirements outgrow what Glue's managed environment allows.